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(54) RECORDING MEDIUM, IMAGE PROCESSING DEVICE, AND IMAGE PROCESSING METHOD 



(57) On a screen display, a specialized image effect 
is easily attained in which near images are made to 
appear clearly while distant images are seen as blurred. 
An image is formed on a screen (200) in which for an 
image (156a M ) pertaining to a background containing 
far-distance object group for which the depth amount 
thereof is comparatively large, the source image is 
made as a semi-transparent processed image for which 
a two pixel portion thereof is shifted in an upward direc- 
tion, for an image (154a') pertaining to a mid-distance 
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object group the source image is made as a semi-trans- 
parent formation processed image for which a one pixel 
portion thereof is shifted in an upward direction, and for 
an image (152a) pertaining to a near-distance object an 
upward shifting processing is not performed. For an 
image after such processing is performed, a near image 
(152a) appears distinctly, a middle-degree distant 
image (154a') appears somewhat blurred and a far-dis- 
tant image (156a") appears still more blurred. 
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Description 
Technical Field 

5 [0001] The present invention concerns a recording medium, an image processing apparatus and method for image 
processing which concerns an effective image display, or image processing, of a three-dimensional image which is dis- 
played an a screen of a CRT display device or the like. 

Background Art 

w 

[0002] With respect to images, there is a "blurring" technique which is used to give a fixed effect of three-dimen- 
sionality or sense of perspective. For example, when a picture is taken with a camera, portions which are in sharp focus 
are photographed distinctly, and accompanying a distancing away from the focus point, the degree of blurring 
increases. It is known that such blurring imparts a sense of perspective to the photograph. 
15 [0003] As a blurring process which is performed by computer processing, there is a technique disclosed, for exam- 
ple, by Japanese Laid-Open Patent Publication 10-74258. 

[0004] This technique is one for which, with respect to data having distance information, image data having a blur- 
ring amount which depends on the distance information is generated by processing the data through a digital low-pass 
filter having a frequency cutoff corresponding to the distance information. 
20 [0005] Notwithstanding, with this technique, because it is required to process image data based on low pass filter- 
ing, such as with use of a digital Butterworth filter, the computational burden for attaining the blurred images is 
extremely large, and accordingly, there is a problem that processing time is required. 

[0006] The present invention, while taking into consideration this type of problem, has an object of providing a 
recording medium, image processing apparatus and method for image processing, which is capable of imparting to a 
25 three-dimensional display image (containing a three-dimensional image therein) a highly superior specialized effect 
heretofore unavailable. 

[0007] Further, the present invention has the object of providing an image processing apparatus and method for 
image processing capable of easily generating a blurred image. 

[0008] Furthermore, the present invention has the object of providing an image processing apparatus and method 
30 for image processing capable of easily generating a screen image (image) to which a specialized effect is imparted 
such that, while items existing in the foreground are not blurred, items which are distanced as compared to items in the 
foreground appear further blurred. 

Disclosure of Invention 

35 

[0009] On the recording medium according to the present invention, a program is stored comprising a step of shift- 
ing, based on depth amount information of a plurality of three-dimensional objects, at least one pixel portion of an image 
on a display which corresponds to an object having large depth amount, from among the plurality of three-dimensional 
objects, and a step of generating a screen image for displaying, on a screen, an image in which the image shifted by 
40 one pixel portion thereof is overlaid on the image existing before such one pixel portion shifting takes place, together 
with an image corresponding to an object having a small depth amount. 

[0010] According to the present invention, from among a plurality of three-dimensional images, a large depth 
amount image can be made as an image having a blurred focus. As a result, when seen from the point of an observer 
(camera), images which are more distant can be seen as blurred images, and images having a small depth amount 
45 (images which are in front of the blurred images), can be seen as images which are in focus (distinct images). Further, 
the pixel number shifted on the display can be selected as an optimal pixel number depending on, for example, the dis- 
play type or the tastes and so forth of the observer. 

[0011] In this case, by rendering an image corresponding to a three-dimensional object having a large depth 
amount on a two-dimensional background image which is rendered to have the largest depth amount, the image display 
so is made more natural. 

[0012] Further, although the direction of the at least one pixel shift on the display can be in any optional direction, 
by making this direction to be either an upward direction or a downward direction on the screen, for example, the inven- 
tion can optimally be applied to an entertainment system (for example, an apparatus having a sound reproduction func- 
tion using a compact disk CD or an image and sound reproduction function using a video CD, and further having a 
55 gaming function) or the like, which uses a horizontal scanning type of display. 

[0013] Herein, by subjecting the blurred image to a semi-transparency formation processing, the blurred image can 
be made as a more harmonious image. 

[0014] Moreover, by making a three-dimensional object to be a fixed and/or moving object, as seen from a view- 
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point created by the image which corresponds to the object, for example when the viewpoint is fixed, a moving object 
which moves from a rearward toward a forward direction of the screen can be changed from a blurred image to a distinct 
image corresponding to the depth amount thereof. Oppositely, an image which moves from a frontal toward a rearward 
direction of the screen can be changed from a distinct image (an image in focus) to a blurred image corresponding to 

5 the depth amount thereof. 

[0015] Further, an image processing apparatus according to the present invention comprises a recording means in 
which a plurality of three-dimensional objects together with depth amount information are stored, a frame buffer having 
a rendering area in which the plurality of three-dimensional objects are converted into images by a fixed process and 
rendered, display means for displaying on a screen images rendered in the rendering area of the frame buffer, and ren- 

10 dering control means for performing conversion processing for converting from the three-dimensional objects to the 
images, wherein the rendering control means shifts, by at least one pixel portion and renders in the frame buffer an 
image corresponding to an object having a large depth amount, from among the plurality of three-dimensional objects, 
based on the depth information of the plurality of three-dimensional objects, generates and draws an image (blurred 
image) for which the image shifted by one pixel portion and rendered is subjected to semi-transparency formation 

15 processing and overlaid on the image existing before the one pixel shifting took place, renders in the rendering area the 
blurred image together with an image having a small depth amount from an image corresponding to the plurality of 
three-dimensional objects, and displays on the display means the blurred image together with the image having a small 
depth amount. 

[0016] The present invention can advantageously be applied to an entertainment system, for example, comprising 

20 the recording means, frame buffer, display means and rendering control means. 

[0017] In this case as well, overlaying onto a two-dimensional image of a farthest distant scene is possible. 
[0018] Further, a control is provided by which the rendering areas are set as two areas in the frame buffer, and 
wherein the rendering control means, at a time when an image rendered in one rendering area is being displayed on 
the screen of the display means, performs rending of an image containing the blurred image in the other rendering area, 

25 and after rending of the image containing the blurred image has been completed, the image rendered in the other ren- 
dering area is displayed on the screen of the display means, wherein as a result of such control, the present invention 
can be applied to a frame buffer having a plurality of rendering areas. 

[0019] In this case as well, the direction of shifting by at least one pixel portion on the screen can be taken as any 
optional direction, however, in the case of a horizontal scan type of display, an upward or downward direction on the 
30 screen is favorable. 

[0020] Naturally, the three-dimensional object can be a fixed object and/or a moving object, as seen from a view- 
point created by the image corresponding to the object. 

[0021] Further, the method for image processing according to the present invention comprises a step of preparing 
source image data in which each pixel value is represented as an RGB value, a step of creating one-pixel shifted image 
35 data for which the source image data is shifted by at least one pixel in a predetermined direction, and a step of creating 
blurred image data formed by overlaying the one-pixel shifted data on the source image data and from RGB values 
respectively added at a fixed ratio to the RGB values of pixels corresponding in position. 
[0022] According to this aspect of the invention, blurred image data can be easily created. 

[0023] In this case, the fixed direction is taken as any of up, down, right or left directions, and by making the fixed 
40 ratio to be roughly 50%:50%, an image in which the focus is shifted with respect to the source image can be easily cre- 
ated. 

[0024] Further, an image processing apparatus according to the present direction comprises: a recording means in 
which a plurality of three-dimensional objects are recorded together with depth information; a frame buffer having a ren- 
dering area in which the plurality of three-dimensional objects are rendered after being subjected to a perspective pro- 

45 jection conversion at a fixed viewpoint and converted into images; display means for displaying on a screen a plurality 
of images which are rendered in the rendering area; and rendering control means for performing the perspective pro- 
jection conversion processing, wherein the rendering control means, in the rendering area of the frame buffer, classifies 
images corresponding to the plurality of three-dimensional objects, based on the fixed viewpoint, into a far-distance 
image group pertaining to the depth information, a mid-distance image group, and a near-distance image group, 

so renders the far-distance image group in the rendering area, renders images (blurred far-distance image group) in the 
rendering area formed by subjecting to semi-transparency formation processing and overlaying the far-distance image 
group which has been shifted by at least one pixel portion in a first fixed direction on the far-distance image group exist- 
ing prior to such shifting, further renders the mid-distance image group in the rendering area, and further, renders in the 
rendering area images (images formed by a double-blurred far-distance image group and a blurred mid-distance image 

55 group) formed by subjecting to semi-transparency formation processing and overlaying the mid-distance image group 
and the far-distance image group which have been shifted by at least one pixel position in a second fixed direction with 
respect to the rending area in which the mid-distance image group and far-distance image group are rendered, on the 
image group existing prior to such shifting, and further renders the near-distance image group in the rendering area. 
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[0025] According to this aspect of the present invention, it becomes possible to simultaneously display on a display 
device a near-distance image group for which blurring processing is not conducted, a blurred mid-distance image group 
which is shifted by at least one pixel behind the near-distance image group, and further, a blurred far-distance image 
group which is shifted by at least two pixels behind the mid-distance image group. By doing so, on the screen of the 
5 display device, for example, a distinct near-distance image group, a slightly blurred mid-distance image group, and a 
further blurred far-distance image group can be observed, and it is possible to observe an image (screen image) which 
is equivalent to a case in which a camera focus is adjusted on a near-distance image group. 

[0026] In this case as well, behind the far-distance object group a two-dimensional background image group can be 
displayed. 

10 [0027] Further, the first and second fixed directions can be the same direction or in different directions, wherein as 
a result, the direction of the blurring can be set at an optional direction. 

[0028] Furthermore, in the case of a horizontal scan type display device, it is favorable if the direction of shifting by 
at least one pixel portion, on the screen, is taken to be a shifting direction in either an upward direction or a downward 
direction. 

15 [0029] Still further, the three-dimensional object can be a fixed object and/or a moving object, as seen from a view- 
point created by the image corresponding to the object. 

[0030] The above-described and other objects, features and advantages shall be made more clear with reference 
to the accompanying drawings in which an example of a preferred embodiment shall be described. 

20 Brief Description of Drawings 

[0031] 

FIG. 1 is a plan view showing the exterior appearance of a video game apparatus to which art embodiment of the 
25 present invention is applied. 

FIG. 2 is a front view showing a structure of a slot part of the video game apparatus main body. 
FIG. 3 is a exterior perspective view of the video game apparatus main body. 

FIG. 4 is a electrical block diagram showing a detailed structural example of a principal part of the video game 
apparatus. 

30 FIG. 5 is a flow chart showing process steps for image processing according to an embodiment of the invention. 

FIG. 6 is a typical view for furnishing an explanation of an aspect by which an object is divided into near-, mid-and 
far-distance object groups. 

FIG. 7 is a line diagram for furnishing an explanation of a rendering area of a frame buffer. 
FIG. 8 is a line diagram for furnishing an explanation of a screen region. 
35 FIG. 9A is a typical view showing a rendered state of a background which is rendered in a rendering area of the 

frame buffer. 

FIG. 9B is a typical view showing a rendered state of images pertaining to a far-distant object group over-written on 
the image of FIG. 9A. 

FIG. 9C is a typical view showing a rendered state of the images after execution of a blurring process, with respect 
40 to the image of FIG. 9B. 

FIG. 9D is a typical view showing a rendered state of images pertaining to a mid-distance object group over-written 
on the one-pass blurred image of FIG 9C. 

FIG. 9E is a typical view showing a rendered state of the images after execution of a blurring process, with respect 
to the image of FIG. 9D. 

45 FIG. 9F is a typical view shoving a rendered state of an images pertaining to a near-distance object group, over a 

composite image made up of a two-pass blurred image and the one-pass blurred image of FIG. 9E. 
FIG. 10 is a flow chart showing steps for blur processing. 

FIG. 11 is a line diagram for facilitating explanation of blur processing of a rendering area of the frame buffer; 
FIG. 12 is a typical view showing a source image prior to blur processing. 
so FIG. 1 3 is a typical view showing state of a blurred image made up of a one-pixel shifted source image overlaid onto 

the source image. 

FIG. 14 is a typical view showing a blurred image. 

FIG. 15 is an expanded view of FIG. 9F, and a typical view showing an example of a screen display containing a 
blurred image according to this embodiment. 
55 FIG. 16 is a typical view showing an example of a screen display which does not contain blurring processing, as a 

comparison corresponding to the screen display example of FIG. 15. 
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Best Mode for Carrying Out the Invention 

[0032] Next, an embodiment of the present invention shall be explained with reference to the drawings. 
[0033] FIG. 1 shows an outer appearance of the structure of a video game apparatus 1 as an example of an enter- 
5 tainment system to which the embodiment of the present invention is applied. 

[0034] The video game apparatus 1 is a device for reading out a program of a game or the like, which is recorded 
for example on an optical disk or the like, and which executes a game or the like in accordance with instructions from a 
user (game player). With execution of the game or the like, primarily the progression of the game, display and sound 
are controlled. 

10 [0035] The main body 2 of the video game apparatus 1 comprises: a disk loading portion 3 accommodated in a 
central part of a casing for which the planar aspect thereof is substantially rectangular shaped, and which receives an 
optical disk, such as a CD-ROM or the like, as a recording medium for supplying an applications program; a reset switch 
4 for optionally resetting a game; a power switch 5; a disk operating switch 6 for manipulating the insertion of the optical 
disk; and, for example, two slot portions 7A, 7B. 

15 [0036] Further, the recording medium for supplying the application program is not limited to an optical disk, but for 
example, it is also acceptable for the applications program to be supplied through a communications link. 
[0037] Respective individual operating devices (also called "controllers") 20, of which there are two in total, can be 
connected in slots 7A, 7B, whereby a competitive game or the like can be conducted by two players. Also, a memory 
card 10, or a portable electronic device 100 serving as a portable information terminal can be inserted into slots 7A, 7B. 

20 [0038] Although in FIG. 1 a structure in which two system slots 7A, 7B are disposed has been shown by way of 
example, the number thereof is not necessarily limited to two systems. 

[0039] The operating device 20 has first and second operating sections 21, 22, a left button 23L and a right button 
23R, a start button 24, and a selection button 25, and furthermore, has operating sections 31, 32 which enable analog 
operations, a mode selecting switch 33 for selection of an operational mode for the operating sections 31 , 32, and a dis- 
25 play indicator 34 for displaying the selected operational mode. A non-illustrated mechanism for imposing vibrations is 
disposed in an internal part of the operating device 20. 

[0040] FIG. 2 shows an aspect of the slots 7A, 7B which are disposed on a frontal surface of the main body 2 of the 
video game apparatus 1. 

[0041] In the present invention, slots 7A, 7B are formed respectively in two levels, wherein memory card insertion 
30 portions 8A, 8B for insertion of a memory card 10 or portable electronic device 100 are disposed in the upper level, and 
controller connection ports flacks) 9A, 9B, to which a connection terminal portion (connector) 26 (see FIG. 1) of the con- 
troller 20 is connected, are disposed in the lower level. 

[0042] The insertion holes (slots) for the memory card insertion portions 8A, 8B are formed to have an elongated 
longitudinal shape in the lateral direction, wherein the corners of both ends of the lower side are rounded as compared 

35 to the corners of both ends of the upper side thereof, forming a structure in which the memory card 10 or portable elec- 
tronic device 100 cannot be inserted in a mistaken orientation. Further, shutters are disposed in the memory card inser- 
tion portions 8A, 8B on an interior part thereof which protect the connecting terminals for attaining election connections. 
[0043] On the other hand, the controller connecting parts 9A, 9B are formed by a structure in which both end cor- 
ners of a lower side of the insertion holes, which are made to have an elongated longitudinal shape in the lateral direc- 

40 tion, are formed to be more rounded when compared with both end corners of the upper side thereof, so that a 
connecting terminal portion 26 of the controller 20 is not connected in a mistaken orientation. Further, the insertion 
holes are made different from the memory card insertion portions 8A, 8B so that the memory card 10 or portable elec- 
tronic device 100 will not be mistakenly inserted therein. 

[0044] FIG. 3 illustrates a state in which a portable electronic device 100 is inserted in the memory card insertion 
45 portion 8A of the a slot section 7A on the frontal plane of the video game apparatus 1 . 

[0045] FIG. 4 is a block diagram showing an example of an outline circuit structure of essential elements of the 
above-described video game apparatus 1 . A display 69, such as a television, is connected to the video game apparatus 
1. 

[0046] The video game apparatus 1 is of a configuration comprising: a control system 50 composed of a Central 
50 Processing Unit (CPU) 51 and its peripheral equipment, etc.; a graphics system 60 composed of a picture processing 
unit (Graphics Processing Unit, GPU) 62 for carrying out rendering with respect to a frame buffer 63, etc.; a sound sys- 
tem 70 composed of a Sound Processing Unit (SPU) for generating music or sound effects, etc.; an optical disk control 
unit 80 for carrying out control of an optical disk 79 on which an applications program is recorded; a communications 
control unit 90 for carrying out control of inputfoutput of data from a memory card 10 for storing game settings, etc. and 
55 signals from the controller 20 to which instructions from a user are input, or input/output of data from a portable elec- 
tronic device 100; and a bus (BUS) or the like to which the respective portions are connected. 

[0047] The above-mentioned control system 50 comprises: a CPU 51; a peripheral equipment control unit 52 for 
carrying out interrupt control or control for Direct Memory Access (DMA) transfer, etc.; a main memory (main memory 
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unit) 53 composed of a Random Access Memory (RAM); and a Read Only Memory (ROM) 54 in which a program mak- 
ing up the so-called operating system, or the like, for carrying management of the main memory 53, the graphics sys- 
tem 60, sound system 70, etc., is stored. Further, concerning the main memory 53 referred to herein, it is to be noted 
that a program on this memory is capable of being executed. 
5 [0048] The CPU 51 executes the operating system stored in the ROM 54 to thereby control the entirety of the video 
game apparatus 1, and constitutes, for example, a 32-bit RISC-CPU. 

[0049] Further, the video game apparatus 1 is adapted so that when power is turned ON, the CPU 51 of the control 
system 50 executes the operating system stored in ROM 54 so that the CPU 51 carries out control of the graphics sys- 
tem 60 and/or sound system 70., etc. Moreover, when the operating system is executed, CPU 51 carries out an initial- 

10 ization of the entirety of the video game apparatus 1 such as confirmation of operations, etc., thereafter to control the 
optical disc control unit 80 to execute an applications program of a game or the like which is recorded on the optical disk 
79 accommodated in a disk receiving section 3 (see FIGS. 1 and 3). Upon execution of the game program or the like, 
the CPU 51 controls the graphics system 60 and/or sound system 70, etc., in accordance with user inputs to control the 
display of pictures and/or the generation of sound effects and music. 

15 [0050] Further, the above-mentioned graphics systems 60 comprises a Geometry Transfer Engine (GTE) 61 for 
carrying out coordinate transform processing which includes a perspective projection conversion, etc., a GPU for car- 
rying out rendering in accordance with rendering instructions from the CPU 51, a frame buffer 63 for storing pictures 
rendered by the GPU 62 and for which, in principle, the image is renewed for each generation of an image switching 
signal of a vertical synchronization signal Vsync, and a picture decoder 64 for decoding picture data which has been 

20 compression-encoded by an orthogonal transform such as discrete cosine transform or the like. In this case, the image 
which is rendered in the frame buffer 63 is made a video output via the GPU 62, and this video output is supplied 
through output terminals to a display means 69 serving as a monitor which is an image display device, for example a 
television. The image which is displayed on the screen of the display (which may include three-dimensional images) is 
renewed at each vertical synchronization signal Vsync. 

25 [0051] The GTE 61 comprises a parallel arithmetic mechanism for executing in parallel, e.g., plural operations, and 
is adapted to have an ability to carry out, at high speed, coordinate transforms (for effecting perspective projection, a 
perspective projection conversion for converting an image of three-dimensional space into an image of two-dimensional 
space is included), light source calculations, and matrix or vector operations, etc. in accordance with operation requests 
from the CPU 51 . More specifically, in the GTE 61 , in the event of an operation for carrying out flat shading to paint, e.g., 

30 a triangular polygon by the same color, about 1,500,000 polygon coordinate operations at a maximum can be carried 
out in one second. Thus, in the video game apparatus 1, load on the CPU 51 is reduced and high speed coordinate 
operations can be carried out. In the present embodiment, the CPU 51, GTE 61 and GPU 62 constitute the rendering 
control means. 

[0052] The GPU 62 carries out rendering operations for polygons, etc., with respect to the frame buffer 63 in 
35 accordance with rendering instructions from the CPU 51. The CPU 51 can carry out approximately 360,000 polygon 
rendering operations at a maximum in one second. 

[0053] Further, the frame buffer 63 is comprised of a so called dual port RAM, and is adapted to have the ability to 
carry out rendering operations from the GPU 62 or transfers from the main memory 53, and perform read-outs for dis- 
play at the same time. The frame buffer 63 has, for example, a capacity of 1 M-bytes, and is handled as a matrix having 

40 1024 pixels in the lateral direction and 512 pixels in the longitudinal direction, each respectively having 16 bits. Moreo- 
ver, in the frame buffer 63, there are provided, in addition to the display area (referred to as a rendering area) which is 
output as a rendering area rendered as a screen image and a video output, a CLUT area in which there is stored a 
Color Look-Up Table (CLUT) to which reference is made when the GPU 62 carries out polygon rendering operations, 
and the like, and a texture area in which there is stored material (i.e. textures) which are inserted (mapped) into the pol- 

45 ygons, etc., which are rendered by the GPU 62 after undergoing a coordinate conversion at the time of the rendering 
operation. Material for a two-dimensional background image of a farthest-most scene in which the sky, clouds and the 
like are displayed is also stored in the texture area. 

[0054] The CLUT area and the texture area are adapted to be dynamically changed in accordance with changes, 
and the like, of the display area. 

50 [0055] Moreover, the above-described GPU 62 is adapted to have an ability to carry out, in addition to the above- 
described flat shading, glow shading for performing an interpolation from the color of a vertex of a polygon, to determine 
the color within the polygon, and texture mapping for attaching a texture stored in the texture area onto the polygon. In 
the case of performing such glow shading or texture mapping, the GTE 61 can carry out approximately 500,000 polygon 
coordinate operations, at a maximum, in one second. 

55 [0056] Further, the picture decoder 64 decodes picture data of a still picture or a moving picture which is stored in 
the main memory 53 by a control from the CPU 51 , to store the same in the main memory 53. 

[0057] Moreover, the reproduced picture data is stored in the frame buffer 63 through GPU 62 so that it can be used 
as the background for a picture rendered by the above-described GPU 62. 
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[0058] The above-described sound system 70 comprises an SPU 71 for generating music or sound effects, etc., on 
the basis of instructions from CPU 51, a sound buffer 72 into which waveform data, or the like, is recorded by the SPU 
71, and a speaker 73 for outputting the music or sound effects, etc., which are generated by the SPU 71. 
[0059] The above-mentioned SPU 71, for example, includes an ADPCM decoding function for reproducing sound 
5 data which has been caused to undergo an ADPCM (Adaptive Differential PCM) processing, e.g., 16 bit sound data is 
processed as a 4 bit difference signal, a reproducing function for reproducing the waveform data stored in the sound 
buffer 72 to thereby generate a sound effect or the like, and a modulation function for modulating the waveform data 
stored in the sound buffer 72 for reproducing the same. 

[0060] By providing such functions, the sound system 70 can be used as a so-called sampling sound source, for 
w generating musical or sound effects and the like, on the basis of waveform data which is recorded in the sound buffer 
72 by an instruction from the CPU 51. 

[0061] The above-mentioned optical disk control unit 80 comprises an optical disk unit 81 for reproducing a pro- 
gram or data, etc., which is recorded on an optical disk 79, a decoder 82 for decoding the program or data, etc., in a 
state in which, for example. Error Correction Codes (ECC) have been added thereto, and a buffer 83 for temporarily 
15 storing data from the optical disk unit 81 to thereby allow a read-out operation of the data from the optical disk 79 to be 
conducted with high speed. A Sub-CPU 84 is connected to the above-described decoder 82. 

[0062] Moreover, as sound data which is recorded on the optical disk 79 and which is read out by the optical disk 
unit 81 , there is so-called PCM data in which sound signals have been caused to undergo an analog/digital conversion, 
in addition to the above described ADPCM data. 
20 [0063] As the ADPCM data, sound data which is recorded in a state in which, for example, a difference of 16-bit 
digital data is represented by 4-bits, is decoded into 16-bit digital data by the decoder 82, and is then supplied to the 
above-mentioned SPU 71, where processing such as digital/analog conversion, and so forth, is implemented by the 
SPU 71 . The sound data, thus processed, is used for the purpose of driving the speaker 73. 

[0064] Further, as PCM data, sound data which is recorded, for example, as 16-bit digital data, after decoding by 

25 the decoder 82, is used for the purpose of driving the speaker 73. 

[0065] Further, the communication control unit 90 comprises a communications controller 91 for carrying out con- 
trol of communications with the CPU 51 through the BUS, and wherein controller connection sections 9A, 9B to which 
a controller 20 for input of instructions from the user is connected, and memory card insertion sections 8A, 8B to which 
a memory card 10 or a portable electronic device 100 may be connected as auxiliary memory units for storing status 

30 data of the game, are disposed in the communications controller 91. 

[0066] The controller 20 which is connected to the controller connection section 9A, 9B includes, for example, six- 
teen designation keys for the purpose of inputting instructions from the user to transmit, at about sixty times per second, 
the state of such designation keys to the communications controller 91 . In addition, the communications controller 91 
transmits the state of the designation keys of the controller 20 to CPU 51. 

35 [0067] Thus, instructions from the user are input to the CPU 51. In response thereto, the CPU 51 carries out 
processing in accordance with the instructions from the user, on the basis of a game program, or the like, which is being 
executed. 

[0068] In this case, for performing reading out of a program, display of pictures, or rendering operations, and the 
like, it is necessary to transfer a large quantity of picture data between the main memory 53. GPU 62, picture decoder 

40 64 and the decoder 82, etc., at high speeds. In view of this, in the video game apparatus 1, there can be carried out a 
so-called DMA transfer, to directly undertake transfer of data between the main memory 53, GPU 62, picture decoder 
64 and the decoder 82, by control from the peripheral device control unit 52 without necessity of intervention by the CPU 
51 as described above. Accordingly, load on the CPU 51 due to data transfer can be reduced, and as a result, high 
speed data transfer can be carried out. 

45 [0069] Moreover, when it is necessary to store status data, etc., of a game being executed, CPU 51 transmits such 
data to the communications controller 91, wherein the communications controller 91 writes the data from the CPU 51 
onto the memory card 10 or portable electronic device 100 which functions as a portable information terminal, and 
which is inserted into the slot of memory card insertion portion 8A or memory card insertion portion 8B. 
[0070] Herein, the memory card 10 comprises a main interface for connection to the video game apparatus 1 , and 

50 a memory interface for input/output of data to and from an internal non-volatile memory. 

[0071] Further, the portable electronic device 100 comprises a main interface for connection to the video game 
apparatus 1, a memory interface for input/output of a program or data to and from an internal non-volatile memory, an 
operating portion having a plurality of operating elements, a display portion 130 formed by a liquid crystal display (LCD) 
or the like, and a window section 140 for performing wireless transmission by a wireless transmission means, for exam- 

55 pie, infrared rays. The shape and dimensions of the connector portion of the portable electronic device 100 are the 
same as those of the memory card 10 used by the video game apparatus 1. 

[0072] A protective circuit for preventing electric breakage is included within the communications controller 91 (see 
FIG. 4). The above-mentioned memory card 10 or portable electronic device 100 are separated from the bus 100, and 



7 



EP 1 056 050 A1 



can be attached or detached in a state wherein the power of the apparatus main body is turned ON. Accordingly, in a 
case in which the memory capacity of the memory card 10 or portable electronic device 100 is insufficient, or the like, 
a new memory card 10 can be inserted without interrupting the power supply of the apparatus main body. As a result, 
a new memory card 10 can be inserted in a state in which game data which is required to be backed up is not lost, and 
5 necessary data can be written into the new memory card 10. 

[0073] Moreover, a parallel I/O interface (PIO) 96 and a serial I/O interface (SIO) 97, are interfaces for connection 
of the memory card 10 and portable electronic device 100 with the video game apparatus 1. 

[0074] Next, with reference to the flow chart shown in FIG. 5, an explanation shall be given concerning an image 
processing which includes a blurring process (also called partial blur processing or focus partial shift processing), of the 
w video game apparatus 1 which is constructed as described above. Further, the flow chart of FIG. 5 is a flowchart for 
supplying an operational explanation in a state in which a part of an applications program stored on the optical disk 79 
is already downloaded into a main memory 53, the applications program is executed in the CPU 51 and a game has 
already been opened by the user (game player). 

[0075] In STEP S1, a field of view of an active camera in a game which is currently under use is detected corre- 
15 sponding to a state of progression of the game. In actuality, there exists a plurality of such cameras in the game, 
wherein a currently valid camera, more specifically the active camera, is detected from a position of the character inside 
the game (i.e. an object of a person, vehicle, or the like). 

[0076] Next, in STEP S2, an object (three-dimensional image) from inside the field of view of the active camera is 
detected. 

20 [0077] More specifically, as illustrated by the planar perspective view of FIG. 6, taking the field of view from the per- 
spective of the active camera to be 8 (the vertical directed field of view [image angle] is 6y, and the lateral directed field 
of view (image angle] is 8x), the objects (three-dimensional objects) which can be observed therein are detected. 
[0078] In STEP S2, taking the screen 200 of the display 69 as the origin Z0 (Z = Z0 = 0 ) of a depth (Z-axis) coor- 
dinate, and corresponding to a depth amount Z with the origin point Z0 serving as a standard, for each of depth 

25 amounts Z1, Z2 and Z3 (Z1 < Z2 < Z3) , objects OBn (n = 0-10) are separated and detected as belonging respectively 
to a near-distance object group (near-distance image group) 152, a mid-distance object group (mid-distance image 
group) 154, or a far-distance object group (far-distance image group) 156. 

[0079] As discussed hereinafter, in FIG. 6 the depth amount Z = Z«» indicates the position of a background BG as 
a two-dimensional image. 

30 [0080] Further, FIG. 6 shows a typical view as seen from an overhead vertical orientation in which the active cam- 
era 150 has a line of sight oriented horizontally, and therefore the vertical line of sight 0y is not shown. 
[0081] Objects OBn, respectively, are three-dimensional objects (according to this embodiment, surface models) 
where, in actuality, each object OBn is divided into polygons wherein a topmost coordinate of the polygons, along with 
the color at the topmost point, are recorded associatively on the optical disk 79, and as necessary, are downloaded and 

35 stored in the main memory 53. From this meaning, the optical disk 79 and main memory 53 may be referred to as a 
three-dimensional object storage means. 

[0082] In FIG. 6, from among the objects OBn (n = 0-11) which are respectively typically drawn, an object OBO, for 
example, a road (here, the running path of a car) is displayed, objects OB1, OB2, OB4, OB6, OB8 and OB11, respec- 
tively, show a car which is travelling in a rearward direction (depth direction) into the screen, object OB3 shows a road- 
40 side tree (tree), and the remaining objects OB5, OB7, OB9 and OB 10 respectively show construction items such as 
buildings and so forth. 

[0083] In this case, for example, observing from the viewpoint 1 51 generated by the image, when the active camera 
is still, objects OBO, OB3, OB5, OB7, OB9 and OB 10 are relatively fixed items, whereas objects OB1, OB2, OB4, OB6, 
OB8 and OB1 1 are relatively moving items. 
45 [0084] In the example of FIG. 6, five objects OBO - OB4 are included in the near-distance object group 152 between 
depth amounts Z0 - Z1 (area length d1), six objects OBO and OB4 - OB8 are included in the Mid-distance object group 
154 between depth amounts Z1 - Z2 (area length d2), and six objects OBO and OB7 - OB11 are included in the far- 
distance object group 154 between depth amounts Z2 - Z3 (area length d3). 

[0085] Herein, it is necessary to keep in mind the point that the object OBO making up the road is included in all of 
so the near-distance object group 152, the mid-distance object group 154 and the far-distance object group 156. Further, 
it is important to keep in mind that object OB4 exists in a straddled state over the near-distance object group 152 and 
the mid-distance object group 154, whereas objects OB7 and OB8 exist in a straddled state over the mid-distance 
object group 154 and the far-distance object group 156. 

[0086] Further, one should bear in mind that it is acceptable for a background BG, such as the sky or the like, to 
55 exist in a position for which the depth amount thereof is infinity (an infinite distance point, wherein the depth amount Z 
is Z = Zoo ). On the screen, clouds which exist in an upper orientation of the sky may also be included in the background 
BG, and it is also acceptable for three-dimensional objects, for example, to be included in far-distance object 156. Of 
course, it is also acceptable for three-dimensional objects to exist between the depth amounts of Z3 and Z«>. Due to the 
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simplified nature of this embodiment, the background BG is a blue sky, having clouds included within the blue sky. The 
images of clouds, when necessary, can also possess a texture in a predetermined region of the frame buffer 63. 
[0087] In this case, as shown in FIG. 7, two rendering areas 202, 204, corresponding to two screen portions of a 
256-pixel x 240-pixel screen 200, which are display regions of the display 69 shown in FIG. 8, are set in the frame buffer 
5 63 which possess a 1023-pixel x 512-pixel memory area. 

[0088] One of the rendering areas 202 of the frame buffer 63, as shown in FIG. 7, is set at an address range (x: 0- 
255, y: 0-239) of the frame buffer 63, and another of the rendering areas 204 is set an another address range (x: 0-255, 
y: 256-495). 

[0089] As a result of being structured in this manner, when an image (which herein is a screen frame) that is ren- 
w dered in one rendering area 204 of the frame buffer 63 is displayed on a screen 200 of the display 69 (at this time, it 
may also be said that the rendering area 204 is a display area 204), an image which is to be displayed next on the 
screen is rendered in the other rendering area 202 of the frame buffer 63. More specifically, two images (two screen 
frames) which are rendered in the rendering areas 202, 204 of the frame buffer 63 are alternately displayed on the 
screen 200. Such alternate switching of screen frames is set so as to be switched upon a first vertical synchronizing 
15 signal Vsync after completion of rendering. 

[0090] During the first processing from the above-mentioned Step S1 to the below-mentioned Step S12, for exam- 
ple, the image (screen frame) which is rendered in the lower side rendering area 204 in FIG. 7, is displayed on the dis- 
play area (also called the screen) 200 of the display 69. 

[0091] At this point, in Step S3, with respect to the other rendering area 202, generation of an image to make up 
20 the next horizontal scan display (screen frame) is initiated. In this case, the GPU 62, referring to a display list (screen 
list for subsequent displays) which is generated by the CPU 51 with assistance of the GTE 61, generates the image. 
[0092] In STEP S4, a background BG, which is a farthest most scene, is rendered with respect to the rendering 
area 202, taking the viewpoint 151 as a standard. 

[0093] FIG r 9A shows typically a rendering area 202 of the frame buffer 63 in which there is written a background 
25 BG image 1 55 as an image in which, on the overall screen, clouds 300 - 303 are floating on a blue sky. If the pixel values 
of the image 155 of the blue sky part thereof are expressed as RGB values, for example, the background BG is written 
in as (R, G, B) = (0. 0, 255) , and for pixel values for the image 155 of the white cloud image 300 - 303 part thereof, the 
background BG is written in as (R, G, B) = (255, 255, 255) . 

[0094] Next, in Step S5, objects OBn which have been classified and detected according to Step S2 are referred to 

30 (see FIG. 6), and from among objects OB0 and OB7-OB1 1 which are included within the far-distance object group hav- 
ing a largest depth amount Z following depth amount Z = Z«> , in order of a farthest (largest depth amount) object, or 
more precisely in order of objects OBn which reside at positions which are farther than a distance terminus (within one 
object, the end part thereof having the largest depth amount Z) of an object OBn, more specifically in the order of object 
OB0, object OB1 1, object OB 10, object OB9, a portion of object OB8 and a portion of object OB7, an image 1561a (i.e. 

35 an image of two-dimensional space which is converted from images in three-dimensional space) which is attained by 
perspective projection conversion of objects OB0, OB1 1, OB10, OB9, a portion of object OB8 and a portion of object 
OB7, respectively, from a position of the viewpoint 151 with the screen 200 as a standard, is overwritten and rendered 
at a position which corresponds to the rendering area 202 in which the image 155 of FIG. 9A is drawn (see FIG. 9B). 
[0095] As understood from FIG. 9B, because of being rendered by overwriting, for example, the cloud image 303 

40 becomes concealed by a backside of the building image 306 which corresponds to object OB10. Further, for image 308 
pertaining to object OB7 and image 310 pertaining to object OB8, only parts from within the depth length of d2 are ren- 
dered. Further, for image 308 pertaining to object OB7 and image 310 pertaining to object OB8, only surfaces (surfaces 
formed by polygons) which are visible from the viewpoint 151 are rendered, and surfaces which are oriented toward the 
depth direction of the screen are not rendered. Further, the surfaces formed by polygons are surfaces of objects OBn 

45 obtained by surface modeling. 

[0096] The image shown by FIG. 9B, for which the image 1561a of the far-distance object group 156 is overwritten 
on the background BG image 155, to simplify understanding, henceforth shall be referred to as the far-distance object 
group image 156a. 

[0097] From inside the far-distance object group 1 56, because the distance terminus of the object OB0 is the depth 
50 amount Z3 and the near terminus is the depth amount Z2, the object OB0 can be judged as being the object having a 
farthest distance terminus from among the objects OBn of the far-distance object group 156. In the order of objects for 
which the distance terminus is farthest, the objects are overwritten and rendered in the rendering area 202. 
[0098] Next, in Step S6, it is judged whether the rendering on rendering area 202 of frame buffer 63, for all objects 
OBn residing in the far-distance object group 156, has been completed. 
55 [0099] If the judgement in Step S6 is affirmative, then the blur processing of Step S7 is executed. 

[0100] FIG. 10 shows a flow chart (sub-routine) of a detailed succession of steps for blur processing, which is exe- 
cuted in Step S7. 

[0101] FIG. 11 shows a pixel arrangement (array) for a rendering area 202 which forms the object of blur process- 
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ing. 

[0102] Rendering area 202 is structured as a 240 line, 256 column matrix, shown by pixels anm = aOO - a239255 
(these also express pixel values, although In actuality each of the pixel values is expressed by a respective RGB value). 
[0103] For blur processing, in Step S101, the pixels a10 - a 1255 of the second line of source pixels P (anm) are 
5 picked up (i.e. the second line of pixels a10 - a1255 is focused on) and each pixel a10 - a 1255 of the picked up (focused 
upon) second row is shifted up by one pixel portion in Step S102. 

[0104] In Step S103, an average is taken between each of pixels a10 - a1255 which have been shifted by one pixel 
portion, and each of the pixels aOO - a0255 of the first line, which confront therewith along the same column (i.e. a semi- 
transparency formation processing is performed). 
10 [0105] In Step S4, each of the pixel values, after having taken such an average, are written as the pixel values for 
the first column pixels a00-a025. 

[0106] For a one-pass processing of steps S101 - S104, for example, at the position of pixel aOO, (aOO + a10)/2 is 
written in as the pixel value. Similarly, at the position of pixel a0255, (a0255 + a1255)/2 is written in as the pixel value. 
[0107] Next, in Step S105, it is judged whether or not the current processing is for the last line (according to this 
15 embodiment, the 239th line), and in the case that it is not the last line, in Step S106, a pointer is referenced and the 
process is advanced by one line. 

[0108] During subsequent processing of steps S101 - S104, pixels a20 - a2255 are overlaid on pixels a10 - a 1255, 
and respectively averaged pixel values {(a 10 + a20)/2 to (a1255 + a2255)/2) } are assigned as the pixel values for pix- 
els a10 - a1255. Similar processing is conducted until the last line of pixels a2390 - a239255 is reached. However, the 

20 values of pixels a2390 - a239255 belonging to the last line are left at their original values. 

[0109] Concerning such blur processing, an explanation shall be given at this point concerning an image a corre- 
sponding to a specified object which is made up from a 4 x 5 block of pixels ct1 1 - a45, as shown in FIG. 12. Herein, for 
ease of understanding, the color (pixel value) of the pixels cc1 1 - a45 of the image a corresponding to the specified 
object are all red (R, G, B) = R(255, a, 0) , and the color (pixel value) of surrounding pixels (not shown) of the image a 

25 are blue (R, G. B) = B(0. 0, 255) . 

[0110] During the blur processing of Steps S101 - S106 shown in FIG. 10, as a result, first, it can be thought of as 
though the entire image a corresponding to the specified object is shifted upward by one pixel and is overlaid upon the 
original image a. 

[0111] FIG. 13 is a typical view showing a drawing of the image a illustrated in FIG. 12 together with an image a' 
30 which has been shifted up by one pixel. 

[0112] Each pixel cc1 1 - ct15 of the first line making up the image a* is overlaid on a blue (R, G, B) = B(0, 0, 255) 
pixel. 

[0113] FIG. 14 shows the pixel values of an image p after averaging processing (semi-transparency formation 
processing) of image a and image a*. In the image p, it is seen that the overall size of the image becomes larger by one 

35 pixel portion only in the upward direction. At this time, the pixel values of the uppermost (first) column are the pixel val- 
ues of red (R, G, B) = R(255. 0, 0) and blue (R, G, B) = B(0, 0, 255) after semi-transparency formation processing, 
and hence become magenta (R, G, B) = M(128, 0, 128) in color; whereas the pixel values of the second through fifth 
lines, also after semi-transparency formation processing, are the semi-transparency processed uniform pixel values, 
and thus, the pixel values are the same as the red pixels of the image a. In this way, for the processed image p, the 

40 image of the first line which is changed in color is perceived as being a pseudo-blurred image. 

[01 14] Of course, the image area of the image a need not be a single red color, but for the case of an image having 
a pattern, for the image p in the case that one-pixel shifting semi-transparency formation processing is performed, not 
only the first line thereof, but the pattern also results in a patterned image having the pattern therein blurred as well. 
[01 15] In general, for the semi-transparency formation processing, when an image of pixels which confront the pix- 

45 els of positions in an image A(R, G, B) are taken as A*(R\ G\ B'), it is understood how to calculate the image B(Rb, Gb, 
Bb) after semi-transparency formation processing in accordance with a mutually-additive averaging by the following 
equation (1). 

B(Rb, Gb, Bb) = B{(R + R')/2, (G + G*)/2, (B + B')/2) 

50 

[0116] Further, in the present embodiment, averaging by such semi-transparency formation processing need not 
be an additive averaging divided by two (50% : 50%), but it is also acceptable to have such division performed by a divi- 
sor other than 2, or an averaging based on applying respective weightings. 

[01 1 7] More specifically, it is also acceptable to perform calculation of the resultant image B(Rb, Gb, Bb) after semi- 
55 transparency formation processing of image A(R, G, B) and image A'(R\ G', B') by means of mutually-additive averag- 
ing as shown by the following equation (2), in which the RGB values are calculated by respectively predetermined 
ratios. Moreover, in equation (2) the value of x is taken as being O^xi 1 . 
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B(Rb, Gb. Bb) = B[{R x x + R'(1— x)}, G x x + G'(1— x)}, (B + B'(1— x)}}] 

[0118] Further, in this embodiment, the direction of pixel shifting is by one pixel in the upward direction, however, it 
is acceptable to be by one pixel in a downward direction as well. In the case that the display 69 employs a television as 

5 the monitor, which performs horizontal scanning and on which an image is formed on the television, because bleeding 
of color tends to be generated in the left and right directions (horizontal direction) of the screen, shifting not in a hori- 
zontal direction, but rather it is preferable to select one of an upward direction (vertical upward direction) or a downward 
direction (vertical downward direction) for which the effectiveness of focus blurring is high. Of course, shifting in any 
optional direction such as up, down, left or right, or in a radiating direction or the like, is possible depending on the dis- 

w play format of the display. 

[01 1 9] Further, as for the number of pixels by which shifting is done, in the one-pass blur processing of this embod- 
iment, a one-pixel shifting is used. This is because, at the resolution (525 line) of current television monitors, for a 
greater than one-pixel shifting, it is highly possible that, rather than focus blurring, an effect of double-copying will occur. 
Of course, an optional predetermined pixel number portion can be determined by experimentation in accordance with 

15 the type of display used. However, it is preferable for shifting to be by at least one one-pixel. 

[0120] In FIG. 9C, an image 156a' in the rendering area 202 after the blur processing of Step S7 is shown. In the 
blurred image 156a', the background BG image and the image 156a of the far-distance object group 156 shown in FIG. 
9B become a one-pixel blurred image in an upward direction. 

[0121] In Step S8, from among objects OB0 and OB4-OB7 that are included in the mid-distance object group 154 
20 having a next largest depth amount Z, a farthest object ordering, or more precisely, an ordering of objects OBn which 

reside at positions which are farther than a distance terminus of an object OBn, more specifically in the order of object 

OB0, object OB7, object OB6 and object OB5, an image (i.e. three-dimensional — > two-dimensional converted image) 

154a which is attained by perspective projection conversion of objects OB0 and OB5 - OB7, respectively, from the view- . 

point 151, is overwritten on the screen 200 and rendered at a position which corresponds to the rendering area 202. 
25 [0122] Further, inside the mid-distance object group 154, for object OB0 although the near terminus thereof is the 

depth amount Z1, the distance terminus thereof is the depth amount Z2, and therefor, from among objects Obn, it is 

judged to be the abject for which the distance terminus thereof is farthest. 

[0123] Next, in Step S9, it is determined whether rendering, on rendering area 202 of frame buffer 63, of all objects 
OBn residing in the mid-distance object group 154 has been completed. 
30 [0124] If the judgement in Step S9 is affirmative, as shown in FIG. 9D, in the rendering area 202, a state is produced 
in which the one-pass blurred image 156a' of the far-distance object group which includes the background image BG, 
and a perspective projection converted image 154a of the mid-distance object group 154 for which blur processing has 
yet to be performed, are written into the rendering area 202. 

[0125] Next, in Step S10, blur processing is executed again. More specifically, with respect to the rendering area 
35 202 in which the image shown in FIG. 9E has been rendered, the blur processing pertaining to the sub-routine steps, 
from Step S111 until Step S1 15, as shown in FIG. 10, is executed. 

[0126] As a result of the fact that additional blur processing is performed on the blurred image 156a' shown in FIG. > 
9D which is rendered in rendering area 202, as shown in FIG. 9E, the blurred image 156a* of the background image BG 
containing far-distance object group 156 becomes a blurred image 156a" which is shifted upwardly further by one pixel 
40 portion, and the one-pass blur processed image 154a' of the mid-distance object group 154, for which image 154a of 
the mid-distance object group 154 has been shifted upwardly by one pixel, becomes rendered over the two-pass blurred 
image 156a". 

[0127] In this manner, as a result of performing blur processing twice, an effect is achieved in which the blurred 
image 156a" pertaining to the far-distance object group 156 appears to be still further blurred (i.e. is seen to be blurred 

45 by one pixel portion more than) than the blurred image 154a' which pertains to the mid-distance object group 154. 

[0128] Next, in Step S11, from among the images 152a of the near-distance object group which make up the 
remaining objects, they are written in order of the farthest object into the rendering area 202 of the frame buffer 63, and 
in Step S12 it is judged that the rendering of images 152a of the near-distance object group 152 has been completed. 
[0129] A state of the rendering area 202 at a point in time at which rendering is completed is shown in FIG. 9F. Fur- 

50 ther, an expanded view of FIG. 9F is shown by FIG. 15. Further, in the images of FIG. 9F and FIG. 15, for explaining 
the effect of blurring, the shifting amount is drawn exaggeratedly in comparison to the actual shifting amount. The actual 
shifting amount is by an upward two-pixel portion for the blurred image 1 56a" of the far-distance object group 1 56 which 
includes the background BG, and an upward one-pixel portion for the blurred image 154a' of the mid-distance object 
group 154. 

55 [0130] Further, in FIG. 15, a code OBna to which an alphabetic character "a" is attached to the object OBn, for 
example a code of OB10a or the like, implies an image corresponding to an object OBn. 

[0131] As described above, in the images of FIGS. 9F and 15, an image (non-blurred image) 152a of the near-dis- 
tance object group 152 made up of a normal, clear image for which blur processing is not carried out, a blurred image 
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154a' of the mid-distance object group for which a one-pixel portion blur processing is carried out, and a blurred image 
156a" of the background BG including far-distance object group for which a two-pixel portion blur processing is carried 
out, are rendered. 

[0132] In Step S13, at the next vertical synchronization signal Vsync, the display of screen 200 is switched, by 
5 switching of the contents rendered in the rendering area 202 with the rendered content of the rendering area 204, which 
is currently being displayed on the screen 200. 

[0133] As discussed above, FIG. 15 is an image that is displayed on the screen 200 of the display 69 following 
switching of the screen display. A user (player) of the video game device who operates the controller 20 etc. can visually 
perceive the image 152a corresponding to the near-distance object group 152 as non-blared, clear image, can visually 

10 perceive the image 154a' corresponding to the mid-distance object group 154 as a somewhat blurred image, and can 
visually perceive the image 156a M corresponding to the background image GD including far-distance object group 156 
as being the most blurred image. By carrying out blur processing in this manner, it is possible to attain an effective 
image which resembles the characteristics of a camera. More specifically, an image (screen image 220) agreeing with 
a camera-like characteristic in which near objects which are in focus are seen clearly, yet further distant objects are 

15 seen as blurred, can be observed on the screen 200. 

[0134] In FIG. 16, as a comparison, an image 222 of the screen 200 is shown for the case in which blur processing 
was not performed. By comparing this image 222 with the image 220 after blur processing shown in FIG. 15, it is under- 
stood that the blur processed image 220 shown by FIG. 15 is superior in terms of the sense of perspective that player, 
or the like, perceives. 

20 [0135] Next, step 1 is returned to again, and an active camera 150 at the current point is judged, and once again in 
accordance with Step S2, objects within the field of view of the camera are detected, and again according to Step S3, 
the next screen frame is created by perspective projection conversion. Hereafter, processing from Step S5 to Step S13 
is repeated. 

[0136] As a result of repeating the processing in this way, blur processing according to the present embodiment is 
25 continued. 

[0137] As discussed in detail above, according to the above-mentioned embodiment, corresponding to a depth 
amount (i.e. depth information) of an object OBn in three-dimensional space including a background image BG, and 
with respect to source image data expressing the aforementioned object OBn, image data obtained by making the 
source image data substantially semi-transparent is shifted in position, overlappingly written upon, and is rendered in 

30 the frame buffer 63. As a result, an effect is achieved in which an object having a shifted focus corresponding to the 
depth information thereof, a so-called blurred image, can be displayed on the screen 200 of the display 69. 
[01 38] In addition, in the case of plural objects OBn, excluding the nearest (in terms of visual perception, that which 
is closest to the player) near-distance object group 152, an amount of shifting in position is made larger in the order of 
objects having a largest depth amount, and is caused to be rendered in the frame buffer 63. As a result, an effect is 

35 achieved whereby objects for which the depths thereof are larger are observed to have their focus shifted (i.e. to be 
blared) more. 

[0139] In this case, for the focus shifting processing according to this embodiment, a complex digital filtering, as in 
the prior technique, is not used; therefore, calculations are simplified and the time required for processing such calcu- 
lations can be shortened. 

40 [0140] Further, in the above-mentioned embodiment, because the program pertaining to the flowcharts shown in 
FIG. 5 and FIG. 10 is originally stored on an optical disk 79, it may be said that the optical disk 79 is a medium storing 
thereon a program which comprises: a step for shifting upwardly by at least one pixel portion on the screen 200 an 
image corresponding to an object having a large depth amount Z, from among a plurality of three-dimensional objects, 
based on depth amount Z information of the plurality of three-dimensional objects; and a step for generating an image, 

45 for displaying an image in which, as necessary, the one-pixel shifted image is subjected to semi-transparency formation 
processing and is overlapped on the image before such one-pixel shifting takes place, together with an image having a 
small depth amount Z. 

[0141] Moreover, in the above-described embodiment, blur processing is performed twice, however, when a televi- 
sion of comparatively low resolution on the degree of 525 scan lines is used as the monitor, it is confirmed that such 
50 two-pass blur processing, in terms of visual perception thereof, is appropriate. In the case of a display having a higher 
resolution, by carrying out the blur processing three times or more, an extremely fine focus blurring process can be car- 
ried out, enabling a further realistic image expression. 

[0142] Further, in the case of shifting by three times or more, there is the case that if shifting in the same direction 
is continued, an unnatural image results. In this case, by following an upward shift, for example, with a leftward shift, 
55 and next a downward shift, and thereafter a rightward shift, a blurring effect is achieved in which the direction of shifting 
there is changed. 

[0143] Still further, in the above-described embodiment, for the screen, a screen 200 on a display 69 is used, how- 
ever in the present invention, there can similarly be applied a case in which the image is displayed on the screen of a 
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theater, or the like. 

[0144] Furthermore, in the above-discussed embodiment, the image of near-distance object group 152 is made to 
be in clear focus, however, it is also possible that, for example, the image of the mid-distance object group 154 can be 
in focus, whereas the image of the far-distance object group 156 and image of the near-distance object group 152 are 
5 made to have a shifted focus. 

[0145] Moreover, the present invention is not limited to the above-described embodiment, but naturally various 
structures therefor can be adopted without deviating from the essence of the present invention. 

Industrial Applicability 

10 

[0146] As described above, according to the present invention, a currently rendered image is taken as a transmis- 
sion source, and an image at a position shifted by at least one pixel therefrom is rendered substantially semi-transpar- 
ently. As a result, it is possible to easily generate an image in which the focus thereof is visually perceptively shifted. 
[0147] Further, by matching together with the element of distance on the screen, for example by repeating the blur 
15 processing multiple times depending on the distance, for far objects, multiple portion pixels which are subjected to 
repeat processing are shifted and rendered, and therefore far objects can be displayed as still further focus blurred 
images. 

[0148] In this manner, unlike prior techniques, an extremely excellent characteristic effect (specifically, an effect in 
accordance with an optical characteristic in which, for example, in the case of a camera focus which is set on near 
20 objects, the near objects are seen sharply, whereas things which are farther away are seen as blurred) can be attained 
for a three-dimensional screen image (containing three-dimensional images therein). 

Claims 

25 1. A recording medium (79) storing a program thereon comprising: 

a step (S1 02) of shifting by at least one pixel portion, based on depth amount information of a plurality of three- 
dimensional objects (OB 10, OB8. etc.), an image (306) on a screen corresponding to an object (OB 10, etc.) 
having a large depth amount from among said plurality of three-dimensional objects (OB10, OB8, etc.): and 
30 a step (S8, S9 : FIG. 9D) of generating an image for displaying on the screen an image (156a) in which the 

one-pixel portion shifted image is overlaid onto the image prior to being shifted by one pixel portion, together 
with an image (154a) corresponding to an object having a small depth amount. 

2. The recording medium storing a program thereon comprising: 

35 

a step (S4) of generating a two-dimensional background image (155) for use on a screen display; 
a step (S1 02) of shifting by at least one pixel portion, based on depth amount information of a plurality of three- 
dimensional objects (OB10, OB8, etc.), an image (306) on a screen corresponding to an object (OB10, etc.) 
having a large depth amount from among said plurality of three-dimensional objects (OB10, OB8, etc.), along 
40 with said background image (155) and on said background image (155); and 

a step (S104, S8, S9 : FIG. 9D) of generating an image for displaying on the screen an image (156a') in which 
said one-pixel portion shifted image on the screen is overlaid onto the image (156a) prior to being shifted by 
one pixel portion, together with an image (154a) corresponding to an object having a small depth amount. 

45 3. The recording medium according to claim 1, wherein a direction of shifting the at least one pixel portion on said 
screen is either of an upward direction or a downward direction on said screen. 

4. The recording medium according to claim 2, wherein a direction of shifting the at least one pixel portion on said 
screen is either of an upward direction or a downward direction on said screen. 

50 

5. The recording medium according to claim 1, wherein said step (S104, S8 S9 : FIG. 9D) of generating an image for 
displaying on the screen an image (156a*) in which said one-pixel portion shifted image on the screen is overlaid 
onto the image (156a) prior to being shifted by one pixel portion, together with an image (154) corresponding to an 
object having a small depth amount, comprises: 

55 

a step (S103, S104, S8, S9 : FIG. 9D) of generating an image for displaying on the screen an image (156a') in 
which said one-pixel portion shifted image on the screen is subjected to semi-transparency formation process- 
ing (S104) and overlaid onto the image (156a) prior to being shifted by one pixel portion, together with an 
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image (154) corresponding to an object having a small depth amount. 

The recording medium according to claim 2, wherein said step (S104, S8, S9 : FIG. 9D) of generating an image for 
displaying on the screen an image (156a 1 ) in which said one-pixel portion shifted image on the screen is overlaid 
onto the image (156a) prior to being shifted by one pixel portion, together with an image (154) corresponding to an 
object having a small depth amount, comprises: 

a step (S103, S104, S8, S9 : FIG. 9D) of generating an image for displaying on the screen an image (156a') in 
which said one-pixel portion shifted image on the screen is subjected to semi-transparency formation process- 
ing (S104) and overlaid onto the image (156a) prior to being shifted by one pixel portion, together with an 
image (154) corresponding to an object having a small depth amount. 

The recording medium according to claim 1, wherein said three-dimensional object (OB10, OB8, etc.) comprises a 
fixed (OB 10, etc.) and/or moving (OB8, etc.) object as viewed from a viewpoint generated by an image correspond- 
ing to said object. 

The recording medium according to claim 2, wherein said three-dimensional object (OB 10, OB8, etc.) comprises a 
fixed (OB 10, etc.) and/or moving (OB8, etc.) object as viewed from a viewpoint generated by an image correspond- 
ing to said object. 

An image processing apparatus, comprising: 

recording means (79, 53) on which a plurality of three-dimensional objects (OB 10, OB8, etc.) together with 
depth information are recorded: 

a frame buffer (63) having a rendering area in which said plurality of three-dimensional objects are converted 
to images according to a predetermined process and rendered: 

display means (69) for displaying on a screen the images which are rendered in said rendering area of said 
frame buffer (63); and 

rendering control means (51, 61, 62) for performing a conversion processing from said three-dimensional 
objects to said images, 

wherein said rendering control means (51, 61, 62) shifts by at least one pixel portion and renders on said free 
buffer, based on depth amount information of said plurality of three-dimensional objects (OB 10, OB8, etc.), an 
image (306) corresponding to an object having a large depth amount from among said plurality of three-dimen- 
sional objects, generates and renders an image (referred to as a blurred image) (156a') in which said one-pixel 
portion shifted and rendered image is subjected to semi-transparency formation processing and overlaid onto 
the image prior to being one-pixel shifted, renders in said rendering area said blurred image (156a*) together 
with an image (154a) having a small depth amount from images corresponding to said plurality of three-dimen- 
sional objects (OB10, OB8, etc.), and displays on said display means said blurred image (156a*) and said 
image having a small depth amount (154a). 

An image processing apparatus, comprising: 

recording means (79, 53) on which intonation of a two-dimensional background image (155) for display on a 
screen, and a plurality of three-dimensional objects (OB 10, OB8) together with depth information are recorded; 
a frame buffer (63) having a rendering area (203, 204) in which said two-dimensional background image (155) 
is rendered, and wherein said plurality of three-dimensional objects (OB10, OB8) are converted to images 
(156a) and rendered on said background image (155); 

display means (69) for displaying on a screen the images which are rendered in said rendering area (203 or 
204) of said frame buffer (63); and 

rendering control means (51, 61, 62) for performing a conversion processing from said three-dimensional 
objects to said images, 

wherein said rendering control means (51 , 61 , 62) shifts by at least one pixel portion and renders on said frame 
buffer (63), based on depth amount information of said plurality of three-dimensional objects (OB10, OB8, 
etc.), an image (156a) corresponding to an object (OB10, etc.) having a large depth amount from among said 
plurality of three-dimensional objects (OB10, OB8, etc.) together with said background image (155), generates 
and renders an image (referred to as a blurred image) (156a') in which said one-pixel portion shifted and ren- 
dered image is subjected to semi-transparency formation processing and overlaid onto the image prior to being 
shifted by one pixel portion, renders in said rendering area (203, 204) said blurred image (156a') together with 
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an image (310) having a small depth amount from images corresponding to said plurality of three-dimensional 
objects (OB10, etc.), and displays on said display means said blurred image (156a') and said image having a 
small depth amount (154a). 

5 11. The image processing apparatus according to claim 9, wherein said rendering region (63) is set as two areas (202, 
204) in said frame buffer, and wherein said rendering control means (51 , 61 , 62) effects a control so that, when the 
image which is rendered in one of said rendering areas (for example, 20) is being displayed on a screen of the dis- 
play means (79), rendering of the image which includes the blurred image (156a') is carried out in the other render- 
ing area (in this case, 204), and after completion of rendering of the image which contains the blurred image 

10 (156a') f the image which is rendered in said other rendering area is displayed on said display means (79). 

12. The image processing apparatus according to claim 10, wherein said rendering region (63) is set as two areas 
(202, 204) in said frame buffer, and wherein said rendering control means (51, 61, 62) effects a control so that, 
when the image which is rendered in one of said rendering areas (for example, 20) is being displayed on a screen 

15 of the display means (79), rendering of the image which includes the blurred image (156a') is carried out in the 

other rendering area (for example, 204), and after completion of rendering of the image which contains the blurred 
image (156a') t the image which is rendered in said other rendering area is displayed on said display means (79). 

13. The image processing apparatus according to claim 9, wherein a direction of shifting the at least one pixel portion 
20 on said screen is either of an upward direction or a downward direction on said screen. 

14. The image processing apparatus according to claim 10, wherein a direction of shifting the at least one pixel portion 
on said screen is either of an upward direction or a downward direction on said screen. 

25 15. The image processing apparatus according to claim 9, wherein said three-dimensional object (OB 10, OB8, etc.) 

comprises a fixed (OB 10, etc.) and/or moving (OB8, etc.) object as viewed from a viewpoint generated by an image 
corresponding to said object. 

16. The image processing apparatus according to claim 10, wherein said three-dimensional object (OB 10, OB8, etc.) 
30 comprises a fixed (OB 10, etc.) and/or moving (OB8, etc.) object as viewed from a viewpoint generated by an image 

corresponding to said object. 

17. An image processing method, comprising: 

35 a step of preparing source image data (a) in which each pixel value therein is expressed as RGB value; 

a step of creating one-pixel shifted image data (a*) in which said source image data (a) is shifted by at least 
one pixel in a predetermined direction; and 

a step of creating blurred image data (p) formed as a result of overlaying said one-pixel shifted image data (a') 
on said source image data (a), and RGB values to which the RGB value of pixels corresponding in position are 
40 added respectively at a predetermined ratio. 

18. The image processing method according to claim 17, wherein said predetermined direction is one of an upward, 
downward, rightward or leftward direction, and said predetermined ratio is 50%:50%. 

45 19. An image processing apparatus, comprising; 

recording means (79, 53) on which a plurality of three-dimensional objects (OB 10, OB9, etc.) together with 
depth information are recorded; 

a frame buffer (63) having a rendering area (202 or 204) in which said plurality of three-dimensional objects 
50 (OB 10, OB9, etc.) are rendered after being subjected to perspective projection conversion from a predeter- 

mined viewpoint and converted into images; 

display means (69) for displaying on a screen a plurality of images which are rendered in said rendering area 
(202 or 204); and 

rendering control means (51, 61, 62) for performing processing of said perspective projection conversion; 
55 wherein said rendering control means (51,61, 62): 

classifies, based on said fixed viewpoint, images corresponding to said plurality of three-dimensional objects 
(OB10, OB8. etc.) in said rendering area (202 or 204) of said frame buffer (63), into a far-distance image group 
(156), a mid-distance image group (154) and a near-distance image group (152), pertaining to said depth infor- 
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mation, and renders said far-distance object group (156) in said rendering area (202); 

renders, in said rendering area (202 or 204), an image (referred to as a blurred far-distance image group) 
(156a') for which a far-distance image group, in which the far-distance object group (156) has been shifted by 
at least one pixel in a first predetermined direction, is subjected to semi-transparency formation processing and 
overlaid on said far-distance image group (156) before being shifted; 
further renders in said rendering area (202 or 204) said mid-distance image group (154); 
further shifts, with respect to the rendering area (202 or 204) in which said mid-distance image group (154) and 
said blurred far-distance image group (156a') are rendered, said mid-distance image group (154) and said 
blurred far-distance image group (156a*) by at least one pixel in a second predetermined direction, and renders 
in said rendering area (202 or 204) an image (an image resulting from a double blurred far-distance image 
group and a blurred mid-distance image group) (see 156a" and 154a' in FIG. 9E) which is subjected to semi- 
transparency formation processing and overlaid on the image group before being shifted; and 
further renders, in the rendering area (202 or 204), said near-distance image group (152). 

20. An image processing apparatus, comprising: 

recording means (79, 53) on which a plurality three-dimensional objects (OB10, OB9, etc.) together with depth 
information are recorded, and in which a two-dimensional background image (155) for a display screen are 
recorded; 

a frame buffer (63) having a rendering area (202 or 204) in which said background image (155) is rendered, 
and in which a plurality of three-dimensional objects (OB 10, OB9, etc.) are rendered after being subjected to 
perspective projection conversion from a predetermined viewpoint and converted into images; 
display means (69) for displaying on a screen a plurality of images which are rendered in said rendering area 
(202 or 204); and 

rendering control means (51, 61, 62) for performing processing of said perspective projection conversion; 
wherein said rendering control means (51, 61, 62): 

classifies, based on said fixed viewpoint, images corresponding to said plurality of three-dimensional objects 
(OB 10, OB8, etc.) in said rendering area (202 or 204) of said frame buffer (63), into a far-distance image group 
(1 56), a mid-distance image group (1 54) and a near-distance image group (1 52), pertaining to said depth infor- 
mation, and renders said far-distance object group (156) in said rendering area (202), after said two-dimen- 
sional background image (155) is rendered; 

renders, in said rendering area (202 or 204), an image (referred to as a blurred far-distance image group) 
(156a') for which a background image attached far-distance image group, in which the far-distance object 
group (156) together with the background image (155) have been shifted by at least one pixel in a first prede- 
termined direction, is subjected to semi-transparency formation processing and overlaid on the background 
image attached far-distance image group (156a) before being shifted; 
further renders in said rendering area (202 or 204) said mid-distance image group (154); 
further shifts, with respect to the rendering area (202 or 204) in which said mid-distance image group (154) and 
said blurred far-distance image group (156a*) are rendered, said mid-distance image group (154a) and said 
blurred far-distance image group (156a') by at least one pixel in a second predetermined direction, and renders 
in said rendering area (202 or 204) an image (an image resulting from a double blurred far-distance image 
group and a blurred mid-distance image group) (see 156a" and 154a* in FIG. 9E) which is subjected to semi- 
transparency formation processing and overlaid on the image group before being shifted; and 
further renders, in the rendering area (202 or 204), said near-distance image group (152a). 

21. The image processing apparatus according to claim 19, wherein said first predetermined direction and said second 
predetermined direction are of a same direction or of a different direction. 

22. The image processing apparatus according to claim 20, wherein said first predetermined direction and said second 
predetermined direction are of a same direction or of a different direction. 

23. The image processing apparatus according to claim 19, wherein a direction of shifting of the at least one-pixel por- 
tion on said screen is either of an upward direction or a downward direction on said screen. 

24. The image processing apparatus according to claim 20, wherein a direction of shifting of the at least one-pixel por- 
tion on said screen is either of an upward direction or a downward direction on said screen. 

25. The image processing apparatus according to claim 19, wherein said three-dimensional object (OB10, OB8, etc.) 



16 



EP 1 056 050 A1 



comprises a fixed (OB10, etc.) and/or moving (OB8, etc.) object as viewed from a viewpoint generated by an image 
corresponding to said object. 

26. The image processing apparatus according to claim 20, wherein said three-dimensional object (OB 10, OB8, etc.) 
5 comprises a fixed (OB 10, etc.) and/or moving (OB8, etc.) object as viewed from a viewpoint generated by an image 

corresponding to said object. 
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